Method and system for gaze-based control of mixed reality content

ABSTRACT

Systems and methods are presented for discovering and positioning content into augmented reality space. A method includes forming a three-dimensional (3D) map of surroundings of a user of an augmented reality (AR) head mounted display (HMD); determining a depth-wise location of a gaze point of a user based on eye gaze direction and eye vergence; determining a visual guidance line pathway in the 3D map; guiding an action of the user along the visual guidance line pathway at one or more identified focal points; and rendering a mixed reality (MR) object along the visual guidance line pathway at a location corresponding to a direction of the user&#39;s gaze.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a national stage application under 35 U.S.C.§ 371 of International Application No. PCT/US2019/027328, entitled“METHOD AND SYSTEM FOR GAZE-BASED CONTROL OF MIXED REALITY CONTENT,”filed on Apr. 12, 2019, which claims benefit under 35 U.S.C. § 119(e)from U.S. Provisional Patent Application Ser. No. 62/660,428, entitled“Method And System For Gaze-Based Control Of Mixed Reality Content,”filed Apr. 20, 2018, each of which is hereby incorporated by referencein its entirety.

BACKGROUND

In the field of mixed reality (MR) head mounted displays (HMDs) orgoggles, there are interaction issues that need to be addressed. Inparticular, HMD users need to manipulate and control mixedreality/augmented reality (MR/AR) objects embedded into an environment.Enhanced control can relate to providing additional information, andpresenting the information in a users field of view, while optimizingthe view with respect to target device specifications, such as focallyaccurate viewing planes, and resolution.

User interface activities required for viewing the details of an AR/MRobject, such as selecting the object, and moving it closer for adetailed view are currently typically done via a combination ofmodalities and gestures, such as selection via head direction, andmanipulation via hand gestures. For example, to move an object closerwith the Microsoft™ HoloLens™, the user has to do the followingsteps: 1. Turn his head towards an “adjust” tool icon on the corner ofthe object to move; 2. Activate the adjust tool by a finger tap; 3, Turnhis head towards a “drag to move” icon that appears on the object; 4.Start moving the object by performing a tap-and-hold finger gesture; 5.Move the object by moving his hand; 6. Release the tap-and-hold fingergesture to drop the object; 7. Turn his head to a “done” icon on theobject; and 8. Perform the finger-tap gesture to activate the “done”icon.

Although the finger tap and hand move gestures can be replaced by usinga handheld gyro-based “clicker” controller, this approach is bothlaborious and conspicuous, and difficult in public locations.

For more natural interaction, the use of eye gaze is widely studied,originally as an input method for disabled people. Gaze tracking systemsare also making their way into AR/MR HMDs; for example, Eyefluence™,owned by Google are building a gaze gesture based system with HMDmanufacturers.

A number of interaction methods using gaze control have been researched,such as Gaze pointing, Gaze gestures, Dwell-based selection, Multimodalselection, Selection by following a moving object, Drag and drop, Rotarycontrol and sliders, Switching windows, Image annotation, Reading, andFocus of attention. HMD-friendly approaches for multimodal objectcontrol have been reported in Evaluation of HeadTurn: An InteractionTechnique Using the Gaze and Head Turns. In Proceedings of the 9thNordic Conference on Human-Computer Interaction (NordiCHI 2016) andEnhanced gaze interaction using simple head gestures. In Proceedings ofthe 2012 ACM Conference on Ubiquitous Computing (UbiComp 2012), withalternative ways of interaction.

Focusing on the function of quickly examining an object, selecting andmanipulating (moving) the object are of particular interest. Thefunction resembles drag and drop using eye gaze, and some approacheshave been described at least in “The use of eye movements inhuman-computer interaction techniques: what you look at is what youget.” ACM Trans. Inf. Syst. 9, 2 (Jacob, April 1991) and Gaze-basedInteraction for Virtual Environments. J. UCS, 14(19), 3085-3098(Jimenez, Gutierrez, D., Latorre, 2008). However, the few reportedstudies focus on 2D actions only, and fail to consider the depth aspect,for example, movement of an object in 3D space, as would be required onan MR HMD device. Also, the approaches require a number of steps akin tothe Hololens™ example above.

As MR content enhancement starts to become commonplace, MR HMD userswill face difficulties in controlling how and when to display thatcontent. Such content enhancement could be associated with practicallyany real-world or virtual content the user sees, such as street lights,traffic signs, shop signs and ads in shop windows, public notices,people, vehicles, etc.

Another problem with current solutions is that some sections in theuser's view are better for displaying content enhancement than others.For example, real-life and virtual objects at different ranges occupythe users view, and the user may be on the move. Further restrictionsmay derive from hardware; gaze recognition and optical displayresolution may set requirements to the area the enhanced content displayrequires. The user cannot be expected to decide each time where in thefull extent of his view to place the content; current systems fail toidentify suitable locations for the content, and show it there, and alsofail to let the user quickly decide the display area.

Another issue is the limited optical capabilities of HMD devices.Current devices, such as the Microsoft Hololens™, have a single fixedfocal plane, which causes a vergence-accommodation conflict with MRobjects that do not reside on that plane. The conflict causes eye strainand slows down the users ability to determine the exact depth locationof the object. HMDs with multiple focal planes are expected to becomecommercially available in the near future. With such devices, a userwould benefit from being able to control the location of the contentenhancement so that it is placed precisely on a focal plane. Currentsolutions lack some kind of a visual guide to focus on in order tocontrol the position (especially depth) of an MR object with gaze alone.

There is a need to provide the user a quick and natural way to controlthe display of MR content enhancement objects, taking use of theknowledge of optically perfect (focally accurate) locations for the HW,with gaze alone.

Systems and methods set forth herein address these issues, and others.

SUMMARY

Systems and methods set forth herein provide embodiments that use gazecontrol to bring enhanced MR content pertaining to faraway objectscloser to the user. The solution uses a vision guideline implemented asan MR object. The guideline contains points that help the user to focushis gaze, placed at depths equivalent to the focally correct viewingdistances supported by device hardware. The enhanced MR content followsthe user's gaze along the line, thus moving the content closer orfarther from the user. The location and dimensions of the line aredetermined by the system, based on HW restrictions and existingreal-life or MR objects in the users view.

One or more embodiments are directed to a method including forming athree-dimensional (3D) map of surroundings of a user of an augmentedreality (AR) head mounted display (HMD); displaying a mixed reality (MR)object in the 3D map including a visual cue that content enhancement isavailable to the user for the object; activating the content enhancementaccording to user input to the HMD with respect to the visual cue;displaying a visual guidance line pathway in the 3D map; guiding anaction of the user along the visual guidance line pathway at one or moreidentified focal points; and rendering the MR object along the visualguidance line pathway at a location corresponding to a direction of theusers gaze.

In one or more embodiments, the activating the content enhancementaccording to user input to the HMD includes user input of one or more ofa gaze, a head gesture, or a gaze dwelling on the content enhancement.

In one or more embodiments, the displaying the visual guidance linepathway in the 3D map includes displaying the one or more identifiedfocal points as a plurality of focal plane indicators at a plurality ofdepths within the 3D map. In one or more embodiments, the rendering theMR object along the visual guidance line pathway at the locationcorresponding to the direction of the users gaze includes moving theenhancement object along the plurality of focal plane indicators at theplurality of depths to enlarge the MR object.

In one or more embodiments, the guiding the action of the user along thevisual guidance line pathway at the one or more identified focal pointsincludes providing the visual cue, wherein the visual cue includes anext suggested action for the user.

In one or more embodiments, the displaying the visual guidance linepathway in the 3D map includes determining a depth-wise location of agaze point of a user based on eye gaze direction and eye vergence.

In one or more embodiments, a method includes forming athree-dimensional (3D) map of surroundings of a user of an augmentedreality (AR) head mounted display (HMD); determining a depth-wiselocation of a gaze point of a user based on eye gaze direction and eyevergence; determining a visual guidance line pathway in the 3D map;guiding an action of the user along the visual guidance line pathway atone or more identified focal points; and rendering a mixed reality (MR)object along the visual guidance line pathway at a locationcorresponding to a direction of the users gaze.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined by gaze tracking of the user.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined according to one or more hardwarerestrictions of the HMD.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined according to a distance of the MRobject.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined by a movement of the user.

In one embodiment, the determining the visual guidance line pathway inthe 3D map is based on the depth-wise location of the gaze point of theuser and available space in the 3D map. In one embodiment, thedetermining the visual guidance line pathway in the 3D map includesforming the visual guidance line pathway to avoid one or more identifiedobjects in the 3D map.

In one embodiment, the determining the visual guidance line pathway inthe 3D map includes altering the visual guidance line pathway accordingto movements of the user, including one or more of a head tilt, a headpitch, a head yaw, and a gesture.

In one embodiment, the determining the visual guidance line pathway inthe 3D map includes altering the visual guidance line pathway accordingto one or more pivot points determined by a user gaze.

In one embodiment, the method also includes determining a number ofpoints along the visual guidance line pathway as a function of availablefocal planes in the 3D map.

Another embodiment is directed to a system including a processor and anon-transitory computer-readable storage medium storing instructionsoperative, when executed on the processor, to perform functionsincluding forming a three-dimensional (3D) map of surroundings of a userof an augmented reality (AR) head mounted display (HMD); determining adepth-wise location of a gaze point of a user based on eye gazedirection and eye vergence; determining a visual guidance line pathwayin the 3D map; guiding an action of the user along the visual guidanceline pathway at one or more identified focal points; and rendering amixed reality (MR) object along the visual guidance line pathway at alocation corresponding to a direction of the user's gaze.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined by gaze trackingof a user.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined according to oneor more hardware restrictions of the HMD.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined according to adistance of the MR object.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined by a movement ofthe user.

In one or more embodiments of the system, the determining the visualguidance line pathway in the 3D map is based on the depth-wise locationof the gaze point of the user and available space in the 3D map.

In one or more embodiments of the system, the determining the visualguidance line pathway in the 3D map includes forming the visual guidanceline pathway to avoid one or more identified objects in the 3D map. Inone or more embodiments of the system, the determining the visualguidance line pathway in the 3D map includes altering the visualguidance line pathway according to movements of the user, including oneor more of a head tilt, a head pitch, a head yaw, and a gesture.

In one or more embodiments of the system, the determining the visualguidance line pathway in the 3D map includes altering the visualguidance line pathway according to pivot points determined by user gaze.

Another embodiment of the system is directed to the non-transitorycomputer-readable storage medium storing instructions operative, whenexecuted on the processor, to perform additional functions includingdetermining a number of points along the visual guidance line pathway asa function of available focal planes in the 3D map.

Another embodiment is directed to a system including a processor and anon-transitory computer-readable storage medium storing instructionsoperative, when executed on the processor, to perform functionsincluding forming a three-dimensional (3D) map of surroundings of a userof an augmented reality (AR) head mounted display (HMD); displaying amixed reality (MR) object in the 3D map including a visual cue thatcontent enhancement is available to the user for the object; activatingthe content enhancement according to user input to the HMD with respectto the visual cue; displaying a visual guidance line pathway in the 3Dmap; guiding an action of the user along the visual guidance linepathway at one or more identified focal points; and rendering the MRobject along the visual guidance line pathway at a locationcorresponding to a direction of the user's gaze.

In one or more embodiments of the system, the activating the contentenhancement according to user input to the HMD includes user input ofone or more of a gaze, a head gesture, or a gaze dwelling on the contentenhancement.

In one or more embodiments of the system, the displaying the visualguidance line pathway in the 3D map includes displaying the one or moreidentified focal points as a plurality of focal plane indicators at aplurality of depths within the 3D map.

In one or more embodiments of the system, the rendering the MR objectalong the visual guidance line pathway at the location corresponding tothe direction of the users gaze includes moving the enhancement objectalong the plurality of focal plane indicators at the plurality of depthsto enlarge the MR object.

In one or more embodiments of the system, the guiding the action of theuser along the visual guidance line pathway at the one or moreidentified focal points includes providing the visual cue, wherein thevisual cue includes a next suggested action for the user.

In one or more embodiments of the system, the displaying the visualguidance line pathway in the 3D map includes determining a depth-wiselocation of a gaze point of a user based on eye gaze direction and eyevergence.

Another embodiment is directed to a method for rendering a visualguidance pathway including forming a three-dimensional (3D) map ofsurroundings of a user of an augmented reality (AR) head mounted display(HMD); determining a depth-wise location of a gaze point of a user basedon eye gaze direction and eye vergence; determining a visual guidanceline pathway in the 3D map; and rendering one or more mixed reality (MR)objects along the visual guidance line pathway at locationscorresponding to a direction of the user's gaze, while avoiding one ormore preexisting objects in the 3D map of the surroundings.

In one or more embodiments of the method, the visual guidance linepathway is placed in determined available space within the 3D map of thesurroundings.

In one or more embodiments of the method, the one or more preexistingobjects include one or more real-world objects and existing MR objects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a solution architecture for gaze-based control of mixedreality (MR) content in accordance with an embodiment.

FIG. 2 depicts a process for activating, controlling and deactivatingthe vision guideline for controlling an MR object with gaze and headmovement in accordance with an embodiment.

FIG. 3 depicts a field of view illustrating determination of a minimumrequired width of a vision guidance line in a user's view in accordancewith an embodiment.

FIG. 4 depicts a graph illustrating variations in gaze control accuracyadapted from conference paper “Toward Everyday Gaze Input: Accuracy andPrecision of Eye Tracking and Implications for Design,” CHI 2017, May6-11, 2017, p. 1125.

FIG. 5 depicts an example of non-linear vision guidance line to preventthe line from clashing with an existing object in accordance with anembodiment.

FIG. 6 depicts an object in foreground with available MR contentenhancements and indications of possible directions wherein a user canpop-up the content enhancement in accordance with an embodiment.

FIG. 7 depicts an activated content enhancement in accordance with anembodiment.

FIG. 8 depicts a user's drawing of content enhancement to the nearestfocal point with his gaze in accordance with an embodiment.

FIG. 9 depicts a view of a user focusing on an original object with adot along a line indicating a next action in accordance with anembodiment.

FIG. 10 depicts another view of a user focusing on an original objectwith a dot along a line indicating a next action in accordance with anembodiment.

FIG. 11 depicts another view of a user focusing on an original objectwith an enlarged object in accordance with an embodiment.

FIG. 12 depicts a method for activating a guiding an object inaccordance with an embodiment.

FIG. 13 shows schematic illustrations of two steps of a method forchoosing and activating an object in accordance with an embodiment.

FIG. 14 depicts another view of a user focusing on an original objectwith an area highlighted along a visual guidance line pathway inaccordance with an embodiment.

FIG. 15 depicts another view of a user in accordance with an embodiment

FIG. 16 depicts another view of a user focusing on an original objectwith a multiple objects shown in a distance in accordance with anembodiment.

FIG. 17 depicts another view of a user focusing on an original objectwith a multiple visual guidance lines shown in accordance with anembodiment.

FIG. 18A depicts a method with schematic diagrams of a user changing thedirection of a visual guidance line before the enhancement object startsto move along the visual guidance line in accordance with an embodiment.

FIG. 18B depicts a method with schematic diagrams continuing theschematic diagrams of FIG. 18B further illustrating the changing ofdirection of the visual guidance line in accordance with an embodiment.

FIG. 19A is a system diagram illustrating an example communicationssystem in which one or more disclosed embodiments may be implemented.

FIG. 19B is a system diagram illustrating an example wirelesstransmit/receive unit (WTRU) that may be used within the communicationssystem illustrated in FIG. 19A according to an embodiment.

DETAILED DESCRIPTION

Embodiments herein provide systems and methods to enable a user ofvirtual and mixed reality devices to decide whether to show contentenhancement, and if so, enables users to very quickly and effortlesslycontrol how much to see, i.e., if the content enhancement seemsinteresting a user can pull it closer for closer inspection, but also beable to reject the enhancement if on closer inspection it proves out tobe uninteresting. In one embodiment, the distance is fully controllableby the user.

In some embodiments disclosed herein, little or no other interactionthan the user's gaze is required to show content enhancement. As will beappreciated, constantly scanning our surroundings for informationanyway, so having to use an additional input method such as a handgesture to pull the content enhancement closer would be cumbersome andcould draw unwanted attention in a crowd.

Referring to FIG. 1, an overview of a system 100 in accordance withembodiments includes several components, including SimultaneousLocalization and Mapping (SLAM)/3D Mapping module 102, Gaze detectionmodule 104, Enhancement View User Interface Module 110, Head GestureDetection module 120, Hardware Information Provider module 130, andEnhanced View Content Service module 140. Enhancement View UserInterface module 110 includes an enhancement view location determinationmodule 112, a visualization module 114 and a control module 116.

The SLAM/3D mapping module 102 maintains a 3D model of the user'ssurroundings, and the user's position within the 3D model, includinghead position and orientation. Embodiments herein include the use of anysuitable technique for maintaining a 3D model of a user's surroundings,such as structured infrared light patterns, stereo cameras, monocularvisual odometry, time-of-flight cameras or the like.

The gaze detection module 104 recognizes the direction of the user'sgaze, including vergence information to determine the depth where theuser is looking. The gaze detection module 104 also determines how longand how comprehensively the user's gaze has dwelled on an object. Dwelltimes for both the original, non-enhanced MR or real-life object, andthe enhanced MR object are detected.

Enhancement View User Interface module 110, includes Enhancement ViewLocation Determination module 112 that establishes potential locationsin the 3D space around the user for the necessary controls for anenhancement view. Criteria may include, e.g., existing real-life and MRobjects, minimum eye movement and gaze detection resolutionrequirements, and the movement of the user. User preferences may also beconsidered, such as a preference to display the enhancement view aboverather than at or below eye level. Also included in the Enhancement ViewUser Interface Module 110 is a Visualization module 114 that renders allmixed reality (MR) content, and in embodiments provide functionalitiesincluding visual cues of directions so that the user may activate thecontent enhancement feature, rendering the vision guideline, guiding theuser's action via highlighting available and recommended next actions,and rendering the enhanced content MR object along the vision guidelineat a location corresponding the user's gaze direction.

Control module 116 within the Enhancement View User Interface (UI)module 110 provides functionalities including obtaining a list ofavailable enhancements near the user (e.g., by querying the optionalEnhanced View Content Service 140). Among the available enhancements,requesting from the Location Determination module 112 which enhancementsare possible to display; and for possible enhancements, detectingenhancement start gestures (gaze dwell and/or other methods known in theart, such as gaze plus head gesture plus hand gesture combinations), aswell as an indicated direction for the enhancement view; activating theenhancement view and controlling the location of the enhancement objectalong the vision guidance line; determining and highlighting suggestednext actions, for example, when the user is moving the enhancementobject closer, the next focally correct snap point on the visionguidance line may be highlighted; using gaze dwell and a “visualconsumption” metric from the Gaze Detection module 104, estimatingwhether the user has paid enough attention to the enhancement object sothat it can be removed from view if necessary.

In one embodiment, the gesture for ending the enhancement view caninclude having a user turn gaze away from the enhancement object, afterthe “visual consumption” metric has been met; having a user turn headaway; having a user move the enhancement object back to its originallocation using his gaze; and/or a hand gesture.

In one embodiment, a hardware information provider delivershardware-based restrictions that are relevant in calculating the spaceneeded for showing a vision guideline, as well as the depths of thefocally accurate display planes so that visual cues (“snap points”) canbe rendered for the user at those planes along the guideline.

Relevant hardware restrictions are, at least, the number of focallyaccurate planes supported by device optics, and gaze detectionresolution. For example, in one embodiment, if the hardware supportsfive focal planes, the guideline shows five corresponding points. If thehardware further supports very accurate gaze tracking, the points mayalmost overlap in the user's view. With lesser gaze tracing accuracy, inan embodiment, the points are farther apart x/y-wise in order to beproperly recognized from each other. Thus, the line in some embodimentsoccupies more space in the users view (in left-right and/or up-downdirections).

FIG. 1 also illustrates head gesture detection via head gesturedetection module 120. In an embodiment, the head gesture detectioncontinuously monitors the users head movements, and reports recognizedhead gestures to the UI control module 116. In an embodiment, thecontrol module 116 may use the information, e.g., in conjunction withsimultaneous gaze direction/dwell information to determine activation,control, and deactivation events for the view enhancement feature.

In one embodiment, an Enhanced View Content Service 140 is an externalservice that provides information of available enhanced MR content fornearby real-life and MR objects.

The Enhanced View Content Service 140 provides a quick and simple way toget a detail view of a faraway MR or real-life object. A user candetermine how near (and thus, how big) the object is allowed in his viewand has full control of an event with no automatic popups. In oneembodiment, to accommodate for hardware restrictions that wouldotherwise disturb a user, when hardware supports only a limited numberof focal planes, the user is shown where those planes are, and given theoption to place content there for optimal viewing.

The indication of focal planes is beneficial to a user because the eyeis quickest to focus on those planes, and only requires a short time tograsp relevant details.

Referring now to FIG. 2, a flow diagram illustrates a process 200 forgaze-based depth control of a mixed reality (MR) object. Specifically,the flow diagram describes the process for activating, controlling anddeactivating MR content enhancements using gaze and head movement.

As shown, FIG. 2 illustrates a process that enables interaction betweenSLAM module 202, enhanced view control module 204, enhanced viewlocation detection module 206, gaze detection module 208, visualizationmodule 210 and head gesture detection module 212. Within SLAM module 202a user position, head motion and 3D mapping 216 is performed. After a 3Dmapping is established, head motion data 218 is transmitted to headgesture detection module 212 where constant gesture recognition 220takes place using head gesture detection module 212.

Next, head position orientation and 3D map 226 are provided to enhancedview control 204, which also receives all available content enhancements228 that may be nearby. Enhanced View Location Detection module 206receives hardware restrictions 222, and then estimates a minimumrequired space for vision guidelines with respect to any hardwarerestrictions 224.

Within Enhanced View Control module 204, potential content enhancementlocations 230 are requested from the Enhanced View Location Detectionmodule 206, along with pertinent information such as 3D map, headposition and orientation and content to be displayed.

Enhanced View Location Detection module 206 estimates each content withrespect to space available for display and, among the list of contentelements 234, provides those that are possible to display within thespace available 232, to the Enhanced View Control module 204.

Next, Enhanced View Control module 204 provides a list of MR enhancementobject locations and any popup directions 236 to Visualization module210, which then renders any visual cues 238.

Next, Gaze Detection module 208 provides gaze and dwell data 240 toEnhanced View Control Module 204, which also receives head gesture data242 from Head Gesture Detection module 212. Within Enhanced View Controlmodule 204, a determination is made as to start an event based on data244 from either Gaze Detection module 208 and/or Head Gesture Detectionmodule 212. Also, Enhanced View Control module 204 determines snappoints from any hardware focal properties 246. Next, Enhanced ViewControl module 204 provides a vision guideline and enhanced content 248which is provided to Visualization module 210. Visualization module 210renders the guideline and enhanced content 250.

Next, Enhanced View control module 204 receives gaze and dwell data 252from Gaze Detection module 208, which is used to determine enhancedcontent position based on gaze direction and determine and highlightpotential next actions 254. Any updated content, position and highlightsare then provided to Visualization module 210 for rendering 258.

Gaze and dwell data 260 is repeatedly received from Gaze Detectionmodule 208, as is head gesture data 262 from Head Gesture Detectionmodule 212.

Enhanced View Control module 204 next determines an end event 264 andprovides any enhanced content end animation 266 for rendering 268 toVisualization module 210.

According to embodiments described herein, the system continuouslyperforms a background content enhancement scan. The system continuouslymonitors the user's location, head and gaze direction in order todetermine whether there are objects in the users vicinity that have MRcontent enhancements that can be brought to the users view using thesystem. The determination can be based on, e.g., geolocation basedsearch to a (remote) database with enhanced object locations.

Embodiments also relate to determining the potential for displayingcontent enhancements. On obtaining information of enhanced objects inthe user's vicinity, the system continuously maintains information aboutwhether and where the enhanced content could be brought to the user'sview. The 3D space around the user that can potentially be used forenhanced content placement may initially contain all the space visibleto the user, or be limited to specific viewing areas only. For example,the area right above and below the user's eye level could be ruled outin some embodiments.

According to some embodiments, whether content enhancements are broughtto a user's view is determined by considering different parameters thatmay reduce the usable enhanced content display area. For example, the 3Dspace around the user is considered. The system performs SLAM todetermine the locations of real-life objects near the user. Locationswith real-life obstacles are ruled out as potential locations fordisplaying enhanced content. Recognized real-life objects may inaddition be tagged as objects that must not be occluded.

Another parameter considered includes MR objects in the user's view. 3Dspace in the user's view already occupied by MR objects can be avoidedaccording to an embodiment. In general, occlusion of existing MR objectsis also avoided. Task, activity or other priority based metrics may,however, be used to determine whether the enhanced content view mayocclude existing MR content, such as for a brief period of time.

Another parameter considered includes eye movement requirements. As theusers eye movement (gaze) is used for controlling the enhanced objectlocation, the system needs to determine the minimum extent of the visionguidance line in the user's view so that gaze detection can distinguishbetween the vision control points. The minimum extent may be determinedby at least the following properties: gaze tracking accuracy, which canbe a hardware restriction, the number of points to distinguish, whichmay correspond to the number focal planes provided by the optics, usermovement, distance to the object that is enhanced, and the like.

After considering the space restrictions, one or more potential pathsfor the visual guideline are determined. The determination may be basedon user preference (e.g. the user may prefer to use the top of his viewfor content enhancements), avoiding object occlusion, etc. The line maybe linear, a spline, an arc, or any other form.

As shown in FIG. 2, the system also determines visual cues of availablecontent enhancement. Objects that have been determined suitable forcontent enhancement in accordance with the process are highlighted tothe user. The highlighting may be, e.g., a visual border or an icon.Specifically, the highlighting may contain an indication of availabledirections for a visual guideline.

Embodiments include different methods for a user to activate contentenhancement of an object in virtual or mixed reality. Different methodsinclude a gaze dwell on the object or a direction indicator, or byperforming a head gesture or hand gesture while a gaze remains fixed onthe object, and other methods known to those of skill in the art withthe benefit of this disclosure. The direction of head movement may beused to select one of several proposed directions for a guidance line.If, for example, a leftward direction is proposed then object activationoccurs by turning the head to the left. Alternatively, after theguidance line becomes visible, head yaw/pitch such as a with gaze stillfixed to the object, could be used to fine-tune the location of theline.

In some embodiments, content enhancement activation displays theselected vision guidance line, showing markers at optimal viewingdistances. The enhanced object comes into view, at or close to thelocation of the original object. In some embodiments, the enhancedobject is fixed to the vision guidance line from a corner so that theobject, the line, and the markers can be seen at all times.

In one or more embodiments, a determined visual guidance line may bedisplayed, but additional markers or snap-to points are not displayedalong a visual guidance line. In such embodiments, snap-to points may bemaintained internally and a snap-to effect for moving and displaying theenhanced content may still be maintained as the users gaze is shiftedalong a visual guidance line, even though corresponding markers are notvisually displayed.

As one of skill in the art with benefit of this disclosure willappreciate, the visual guidance line can be optional. For example, inone or more embodiments, a visual guidance line is not displayed butadditional markers or snap-to points are displayed, thereby making thevisual guidance line effectively invisible. Thus, the function of movingthe enhanced content from point to point along the visual guidance linechanges in response to the user's gaze as if the visual guidance line ispresent but not displayed.

In other embodiments, neither the determined visual guidance line, northe markers or snap-to points are displayed. Instead, the function ofmoving the enhanced content from point to point along a determinedvisual guidance line in response to changes in a user's gaze isperformed even though the guidance line and associated markers orsnap-to points are not visible to the user. In one or more embodiments,a reduced set of markers or snap to points may be displayed to give theuser a minimal visual cue for moving the enhanced content. For example,markers or snap-to points that are adjacent to a current position of theenhanced content may be displayed adaptively as the enhanced content ismoved so that a user has a minimal visible indication of where contentmay be moved next using a shift in user gaze.

In some embodiments, the vision guidance line is fixed with respect tothe pivot point of the users head/neck. Thus, if the user moves his headas opposed to yaw/pitch, the guidance line moves along, with the originof the line fixed to the source (object that is being enhanced).

One or more embodiments include depth control. In some embodiments, theprocess enables a user to control the depth-wise location of theenhancement object with a gaze by different methods. In one embodiment,the system recognizes that the user's gaze is within predefined boundsfrom the vision guidance line, and thus should be used to control theposition of the enhancement object. The users gaze direction and eyevergence are used to determine the location of the enhancement objectalong the line. Optionally, the accommodation of the user's eye may beused in addition to determine the focal depth related to what the useris looking at.

In one embodiment, the enhancement object is moved to the correspondinglocation on the guidance line. The next optimal viewing location ishighlighted on the line, to encourage the user to move the object tothat location. At optimal viewing locations, a magnetism or snap effectmay be used to keep the object at those locations, which may requireextra eye movement to move past that point.

In one embodiment, depth control of the object stops when the gazeleaves predefined bounds, such as when a user looks at the objectinstead of the line. Depth control may continue if the user looks backat the line.

In one embodiment, the system keeps track of how long and/or howintensively a user has looked at enhanced content to determine whetherthe content can be discarded once the user looks away. Otherwise, normalquick movements of the head such as glancing at a honking car couldunintentionally hide the enhanced content.

In one embodiment, ending display of enhanced content can be performedvia a gesture, such as fixing gaze on a point on the enhanced object,and turning head towards the far end of the vision guidance line. Inanother embodiment, ending display occurs by a user turning the headand/or a gaze away from the enhanced object after content timeout.

Other methods of ending display of enhanced content include a user usinggaze to move the enhanced object back to starting point using the visionguidance line or a hand gesture.

According to one or more embodiments, Visualization module 210 of FIG. 2renders enhanced content, including a vision guidance line, whichrequires determining space required for displaying the line. TheEnhanced View Location Detection module 206 shown in FIG. 2 determineswhether the necessary controls for MR content enhancement can be drawnin the user's view, which can occur at all times, continuously oraccording to system requirements. Thus, in some embodiments, the module206 determines the minimum required dimensions for the line in theuser's view, and any real or virtual objects the line needs to avoid.

The minimum dimensions for the vision guidance line are determinedaccording to embodiments by determining the number of guidance points todraw. The number of guidance points may be the total number of focallycorrect planes supported by a device's hardware, a subset of the planes,or, if the number of planes is low, the list of points may containinterpolated points to provide enough guidance points. In oneembodiment, the interpolated points may be shown differently than thepoints corresponding to focally correct planes.

The minimum spacing between points according to one embodiment isaccording to the hardware needed by device gaze tracking to be able todetect each point. Further, minimum spacing can be affected by otherfactors such as additional movement to the user's head caused bymovement, or other environmental factors.

Referring to FIG. 3, a method for determining a required width of thevision guidance line in the user's view is illustrated. As shown, FIG. 3shows an illustration of ten focally correct planes 306 in two differentuser views (302) and (304). As shown, the original location of theenhancement object is at the location furthest from the user. FIG. 3shows in (302) a gaze detection resolution of 4 degrees 308, meaningthat the total control line 310 must occupy approximately 40 degrees ofeye movement in the user's view and spaced points along the controlline. FIG. 3 shows in (304), a 2 degree resolution 312, with the controlline 314 that covers half of a user view. In FIG. 3, the movement issolely on the horizontal axis; by using both horizontal and verticalmovement, the extremities of the user's eye movements can be reduced.

Referring now to FIG. 4, a graph illustrates variations in gaze controlaccuracy. The graph is adapted from conference paper, “Toward EverydayGaze Input: Accuracy and Precision of Eye Tracking and Implications forDesign,” Feit et al., CHI 2017, May 6-11, 2017, p. 1125. As shown, thegraph demonstrates that gaze control accuracy can vary for differentreasons, such as viewing angle. Embodiments herein accommodate for theissues shown in FIG. 4. For example, the resolution of gaze controlaccuracy may also vary in different parts of the field of view along anx-coordinate 402 and a y-coordinate 404. Shown are size raw 406, sizefiltered 408, target points 410, and screen size 412 and border line420.

In accordance with an embodiment of a method, after the minimum lengthfor the guidance line has been established, candidate paths for the linein the users view are determined. The origin, such as a far end, of thepath is at the object to be enhanced, and the near end is fixed inrelation to the user's head. Path determination considers the 3D spaceoccupied by real-world objects and existing MR objects in the user'sview so that the guidance line does not clash with existing objects.Further criteria for choosing a path for the vision guidance line mayinclude occlusion, such as the line avoids occluding some MR orreal-life objects, and the users movement such that the line is drawn inthe direction the user is moving to maintain his gaze in that generaldirection.

Referring now to FIG. 5, one embodiment is directed to bending theguidance line 504 of a user 502 to prevent it from clashing with anexisting object 510. For example, if a straight line from start tofinish would clash with the real-world or MR object location, bendingwould be appropriate. FIG. 5 also illustrates a user view 500 showing 2degrees of separation 520 between view lines, and how the guidance linecan occupy more sideways directed area than would be strictly necessaryto make a view less crowded or avoid clashing with an object. Analternative would be to draw the line in another direction entirely or,resolution permitting, drawing the line narrower in the user's view sothat the line and an existing object can be shown side by side.

In some embodiments, a user 502 can control the location of theenhancement object with gaze by focusing on visible points of reference.In one embodiment, the visible points of reference are presentregardless of whether the system has a limited or unlimited number offocal planes. In one embodiment, markers along the vision guidance linehelp focus on a next location instead of having to glide the focuscontinuously up and down the line.

In one embodiment, as the line by design covers a large distance indepth, all but the currently focused section are more or less out offocus. To prevent a next location on the line to focus on from beingdifficult to find, or requiring longer focus than necessary, the nextlocation may be highlighted in sequence, to give a visual stimulus inthe users peripheral vision.

Referring now to FIG. 6, the method is illustrated by a view 600 of auser including a visual cue that MR content enhancement is available foran object 602. The system has already determined potential directionsfor the content enhancement view pop-up, indicated by the arrowheads.The user may either gaze at one of the arrowheads, or use some othergesture known in the art, such as a combination of gaze and head/handgestures to activate the content enhancement in either direction.

FIG. 7 illustrates a next stop in the method, specifically, a useractivating an enhancement feature to the right. The enhanced content, a“calling card” 702, is shown, as well as the available focal points 704and 706. To guide the user's actions, the next suggested action ishighlighted with a focal point indicator. The enhanced content is shownat the farthest available focal point. The user can now move theenhanced content along the depth plane by looking at the differentpoints 704 and 706 along the vision guideline. The next suggestedaction, which may be at a different depth and thus out of focus, ishighlighted to give the user a peripheral vision cue of its location. InFIG. 7, the suggested action is to look at the next closer focal plane.

FIG. 8 illustrates a user view after the user has drawn the enhancementobject 800 to the closest focal point with a gaze. In an embodiment, auser can deactivate the enhanced content display 800 by turning hishead, looking elsewhere after a short while has elapsed, or by movingthe object back to starting point with his gaze.

Referring now to FIG. 9, in one embodiment, an enhancement object 900 isillustrated. In some embodiments, as a user focuses on a certain depth,other areas of a scene can be out of focus as shown by the dimming ofthe user view. Thus, the user benefits from seeing a change inperipheral vision, highlighting the next focal point 901.

Referring now to FIG. 10, further examples of a user view include a nextaction displaying a next action as a user focuses on enhancement object1002. FIG. 10 illustrates that only an enhancement object 1002 may be infocus to assist a user in identifying where to look at next, such asfocal point 1003. Thus, FIG. 10 demonstrates that as the user focuses onenhancement object 1002, he naturally cannot see the surroundings veryclearly as they may be out of focus. However, by showing the suggestednext action 1003 (next location to look at) clearly enough, a usershould be able to identify the next action even if it is out of focus.FIG. 11 illustrates further eye movement from enhancement object 1100 tonext action 1101.

Referring now to FIG. 12, activation and control steps are illustrated.As shown in block 1210, a real-life object 1204 or an existing MR objectshows a visual cue 1206 that content enhancement (pop-up) is availablefor that object to user 1202. The user then activates in block 1220, theenhancement by, for example, a combination of gaze and head gesture, orgaze dwell on the cue. The enhancement object 1208 comes into view,along with the vision guideline 1210 that shows focal plane indicators1212 at their corresponding depths. Then, in block 1230, as the usermoves his gaze along the vision guideline 1232, the enhancement objectmoves correspondingly, effectively bringing the object closer forinspection. In blocks 1220 and 1230, FIG. 12 illustrates visual hints ofa next suggested action including next focal plane indicators atdifferent depths.

As shown, mixed reality enhancement available visual cue is presented toa user in block 1210. A gaze determines which object to enhance andavailable directions for enhancement can be determined automatically andshown with a visual cue. Next, in block 1220, activation occurs by agaze dwell or other user input and a guideline with focal planeindicators along with an enhanced mixed reality object appears. Next, inblock 1230, a suggested next action is highlighted with a gaze, such asa position control gaze and a next action may be highlighted.

Referring now to FIG. 13, an alternative activation method illustratesthat in 1310 a user can lock a gaze 1302 on the object to enhance,followed by, shown in 1320, the user turning his/her head 1322 in thedirection of the direction cue 1304 which triggers the generation anddisplay of guidance line 1324. Thus, activation occurs by a gaze lockand head turn. Furthermore, if multiple direction cues are available(see, for example the multiple directional cues 602 in FIG. 6), thedirection of the users head turn may be used to select one of themultiple direction cues (e.g. the directional cue corresponding mostclosely to the direction of the users head turn).

Referring now to FIG. 14, an example of a curved surface is shown.Instead of a line, the guidance helper could be displayed as atwo-dimensional (2D) surface, such as the inner surface of a cone 1402(plain, hyperbolic, tapered, . . . ), a series of concentriccircles/triangles/rectangles, or the like.

According to an embodiment, providing a guidance helper enables a userto have a greater variety for controlling the position of the enhancedobject with eyes only. In one embodiment, focally optimal depths canassist a user. Thus, a user has control over the vertical and horizontalpositioning of the object. The area available for placing theenhancement object may be determined by a user with head movements, suchas tilt, pitch and yaw. As shown, a user sees an area as if looking frominside of a cone 1402. Lines are drawn along the focally optimaldistance. Also shown is a line that acts as a guide for a next suggestedaction.

Thus, a user could change the location of the vision guidance line afteractivation by, for example, head movement or by some other gesture,either freely or between available locations. For example, if the useractivated the vision guidance line to his right, a large enough headtilt upwards would move the vision guidance line to another availablelocation at the top of his field of view. With free movement, head pitchand yaw could move the guidance line up/down and left/right,respectively.

In one embodiment, the system and method provides a method for choosingthe element to enhance, for example, if the number of potential objectsis more than one or if there are objects in a distance.

Referring to FIG. 15, potential enhancement objects are illustrated in awindow 1500. A user can choose which object to activate by gazing or byother user input.

FIG. 16 illustrates activating an object 1600 and bringing the object toa furthest focal plane.

FIG. 17 illustrates a different vision guideline 1704 and 1706 for eachobject 1600. Thus, in one embodiment, a gaze draws an object 1702 nearerfor closer examination.

The user could also be allowed to freely change the direction of theguideline before the enhancement object starts to move along it, asillustrated in FIGS. 18A and 18B. Thus, in block 1802, a user 1806 gazedetermines an object 1804. In block 1810, activation, a guideline 1812appears. Referring to FIG. 18B, in block 1820, turning of a user head1826 with a gaze fixed can turn the guideline 1824 along a pivot point1822 that is either visible or invisible, either up or down or left toright, as shown in block 1830 with a user 1836 turning a gaze to theleft and altering guideline 1832.

Thus a gaze and head gesture can modify the vision guidance line priorto activating a gaze-based depth control. In one embodiment, afteractivation, as long as the user maintains his gaze on the originalobject, head motions control the position of the vision guidance line.Then, after the line is where the user wants it, the line would be fixedin place by moving gaze from the original object onto some point alongthe line.

EXAMPLE NETWORKS FOR IMPLEMENTATION OF THE EMBODIMENTS

FIG. 19A is a diagram illustrating an example communications system 1900in which one or more disclosed embodiments may be implemented. Thecommunications system 1900 may be a multiple access system that providescontent, such as voice, data, video, messaging, broadcast, etc., tomultiple wireless users. The communications system 1900 may enablemultiple wireless users to access such content through the sharing ofsystem resources, including wireless bandwidth. For example, thecommunications systems 1900 may employ one or more channel accessmethods, such as code division multiple access (CDMA), time divisionmultiple access (TDMA), frequency division multiple access (FDMA),orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailunique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM(UW-OFDM), resource block-filtered OFDM, filter bank multicarrier(FBMC), and the like.

As shown in FIG. 19A, the communications system 1900 may includewireless transmit/receive units (WTRUs) 1902 a, 1902 b, 1902 c, 1902 d,a RAN 104/1913, a CN 106/1915, a public switched telephone network(PSTN) 1908, the Internet 1910, and other networks 1912, though it willbe appreciated that the disclosed embodiments contemplate any number ofWTRUs, base stations, networks, and/or network elements. Each of theWTRUs 1902 a, 1902 b, 1902 c, 1902 d may be any type of deviceconfigured to operate and/or communicate in a wireless environment. Byway of example, the WTRUs 1902 a, 1902 b, 1902 c, 10902 d, any of whichmay be referred to as a “station” and/or a “STA”, may be configured totransmit and/or receive wireless signals and may include a userequipment (UE), a mobile station, a fixed or mobile subscriber unit, asubscription-based unit, a pager, a cellular telephone, a personaldigital assistant (PDA), a smartphone, a laptop, a netbook, a personalcomputer, a wireless sensor, a hotspot or Mi-Fi device, an Internet ofThings (IoT) device, a watch or other wearable, a head-mounted display(HMD), a vehicle, a drone, a medical device and applications (e.g.,remote surgery), an industrial device and applications (e.g., a robotand/or other wireless devices operating in an industrial and/or anautomated processing chain contexts), a consumer electronics device, adevice operating on commercial and/or industrial wireless networks, andthe like. Any of the WTRUs 1902 a, 1902 b, 1902 c and 1902 d may beinterchangeably referred to as a UE.

The communications systems 1900 may also include a base station 1914 aand/or a base station 1914 b. Each of the base stations 1914 a, 1914 bmay be any type of device configured to wirelessly interface with atleast one of the WTRUs 1902 a, 1902 b, 1902 c, 1902 d to facilitateaccess to one or more communication networks, such as the CN 1906/1915,the Internet 1910, and/or the other networks 1912. By way of example,the base stations 1914 a, 1914 b may be a base transceiver station(BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NRNodeB, a site controller, an access point (AP), a wireless router, andthe like. While the base stations 1914 a, 1914 b are each depicted as asingle element, it will be appreciated that the base stations 1914 a,1914 b may include any number of interconnected base stations and/ornetwork elements.

The base station 1914 a may be part of the RAN 1904/1913, which may alsoinclude other base stations and/or network elements (not shown), such asa base station controller (BSC), a radio network controller (RNC), relaynodes, etc. The base station 1914 a and/or the base station 1914 b maybe configured to transmit and/or receive wireless signals on one or morecarrier frequencies, which may be referred to as a cell (not shown).These frequencies may be in licensed spectrum, unlicensed spectrum, or acombination of licensed and unlicensed spectrum. A cell may providecoverage for a wireless service to a specific geographical area that maybe relatively fixed or that may change over time. The cell may furtherbe divided into cell sectors. For example, the cell associated with thebase station 1914 a may be divided into three sectors. Thus, in oneembodiment, the base station 1914 a may include three transceivers,i.e., one for each sector of the cell. In an embodiment, the basestation 1914 a may employ multiple-input multiple output (MIMO)technology and may utilize multiple transceivers for each sector of thecell. For example, beamforming may be used to transmit and/or receivesignals in desired spatial directions.

The base stations 1914 a, 19914 b may communicate with one or more ofthe WTRUs 1902 a, 1902 b, 1902 c, 1902 d over an air interface 1916,which may be any suitable wireless communication link (e.g., radiofrequency (RF), microwave, centimeter wave, micrometer wave, infrared(IR), ultraviolet (UV), visible light, etc.). The air interface 1916 maybe established using any suitable radio access technology (RAT).

More specifically, as noted above, the communications system 1900 may bea multiple access system and may employ one or more channel accessschemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. Forexample, the base station 1914 a in the RAN 1904/1913 and the WTRUs 1902a, 1902 b, 1902 c may implement a radio technology such as UniversalMobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA),which may establish the air interface 1915/1916/1917 using wideband CDMA(WCDMA). WCDMA may include communication protocols such as High-SpeedPacket Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may includeHigh-Speed Downlink (DL) Packet Access (HSDPA) and/or High-Speed ULPacket Access (HSUPA).

In an embodiment, the base station 1914 a and the WTRUs 1902 a, 1902 b,1902 c may implement a radio technology such as Evolved UMTS TerrestrialRadio Access (E-UTRA), which may establish the air interface 1916 usingLong Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/orLTE-Advanced Pro (LTE-A Pro).

In an embodiment, the base station 1914 a and the WTRUs 1902 a, 1902 b,1902 c may implement a radio technology such as NR Radio Access, whichmay establish the air interface 1916 using New Radio (NR).

In an embodiment, the base station 1914 a and the WTRUs 1902 a, 1902 b,1902 c may implement multiple radio access technologies. For example,the base station 1914 a and the WTRUs 1902 a, 1902 b, 1902 c mayimplement LTE radio access and NR radio access together, for instanceusing dual connectivity (DC) principles. Thus, the air interfaceutilized by WTRUs 1902 a, 1902 b, 1902 c may be characterized bymultiple types of radio access technologies and/or transmissions sentto/from multiple types of base stations (e.g., a eNB and a gNB).

In other embodiments, the base station 1914 a and the WTRUs 1902 a, 1902b, 1902 c may implement radio technologies such as IEEE 802.11 (i.e.,Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperabilityfor Microwave Access (WiMAX)), CDMA2000, CDMA2000 1×, CDMA2000 EV-DO,Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), InterimStandard 856 (IS-856), Global System for Mobile communications (GSM),Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and thelike.

The base station 1914 b in FIG. 19A may be a wireless router, Home NodeB, Home eNode B, or access point, for example, and may utilize anysuitable RAT for facilitating wireless connectivity in a localized area,such as a place of business, a home, a vehicle, a campus, an industrialfacility, an air corridor (e.g., for use by drones), a roadway, and thelike. In one embodiment, the base station 1914 b and the WTRUs 1902 c,1902 d may implement a radio technology such as IEEE 802.11 to establisha wireless local area network (WLAN). In an embodiment, the base station1914 b and the WTRUs 1902 c, 1902 d may implement a radio technologysuch as IEEE 802.15 to establish a wireless personal area network(WPAN). In yet another embodiment, the base station 1914 b and the WTRUs1902 c, 1902 d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000,GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell orfemtocell. As shown in FIG. 19A, the base station 1914 b may have adirect connection to the Internet 1910. Thus, the base station 1914 bmay not be required to access the Internet 1910 via the CN 1906/1915.

The RAN 1904/1913 may be in communication with the CN 1906/1915, whichmay be any type of network configured to provide voice, data,applications, and/or voice over internet protocol (VoIP) services to oneor more of the WTRUs 1902 a, 1902 b, 1902 c, 1902 d. The data may havevarying quality of service (QoS) requirements, such as differingthroughput requirements, latency requirements, error tolerancerequirements, reliability requirements, data throughput requirements,mobility requirements, and the like. The CN 1906/1915 may provide callcontrol, billing services, mobile location-based services, pre-paidcalling, Internet connectivity, video distribution, etc., and/or performhigh-level security functions, such as user authentication. Although notshown in FIG. 19A, it will be appreciated that the RAN 1904/1913 and/orthe CN 1906/1915 may be in direct or indirect communication with otherRANs that employ the same RAT as the RAN 1904/1913 or a different RAT.For example, in addition to being connected to the RAN 1904/1913, whichmay be utilizing a NR radio technology, the CN 1906/1915 may also be incommunication with another RAN (not shown) employing a GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

The CN 1906/1915 may also serve as a gateway for the WTRUs 1902 a, 1902b, 1902 c, 1902 d to access the PSTN 1908, the Internet 1910, and/or theother networks 1912. The PSTN 1908 may include circuit-switchedtelephone networks that provide plain old telephone service (POTS). TheInternet 1910 may include a global system of interconnected computernetworks and devices that use common communication protocols, such asthe transmission control protocol (TCP), user datagram protocol (UDP)and/or the internet protocol (IP) in the TCP/IP internet protocol suite.The networks 1912 may include wired and/or wireless communicationsnetworks owned and/or operated by other service providers. For example,the networks 1912 may include another CN connected to one or more RANs,which may employ the same RAT as the RAN 1904/1913 or a different RAT.

Some or all of the WTRUs 1902 a, 1902 b, 1902 c, 1902 d in thecommunications system 1900 may include multi-mode capabilities (e.g.,the WTRUs 1902 a, 1902 b, 1902 c, 1902 d may include multipletransceivers for communicating with different wireless networks overdifferent wireless links). For example, the WTRU 1902 c shown in FIG.19A may be configured to communicate with the base station 19914 a,which may employ a cellular-based radio technology, and with the basestation 1914 b, which may employ an IEEE 802 radio technology.

FIG. 19B is a system diagram illustrating an example WTRU 1902. As shownin FIG. 19B, the WTRU 1902 may include a processor 1918, a transceiver1920, a transmit/receive element 1922, a speaker/microphone 1924, akeypad 1926, a display/touchpad 1928, non-removable memory 1930,removable memory 1932, a power source 1934, a global positioning system(GPS) chipset 1936, and/or other peripherals 1938, among others. It willbe appreciated that the WTRU 1902 may include any sub-combination of theforegoing elements while remaining consistent with an embodiment.

The processor 1918 may be a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs)circuits, any other type of integrated circuit (IC), a state machine,and the like. The processor 1918 may perform signal coding, dataprocessing, power control, input/output processing, and/or any otherfunctionality that enables the WTRU 1902 to operate in a wirelessenvironment. The processor 1918 may be coupled to the transceiver 1920,which may be coupled to the transmit/receive element 1922. While FIG.19B depicts the processor 1918 and the transceiver 1920 as separatecomponents, it will be appreciated that the processor 1918 and thetransceiver 1920 may be integrated together in an electronic package orchip.

The transmit/receive element 1922 may be configured to transmit signalsto, or receive signals from, a base station (e.g., the base station 1914a) over the air interface 1916. For example, in one embodiment, thetransmit/receive element 1922 may be an antenna configured to transmitand/or receive RF signals. In an embodiment, the transmit/receiveelement 1922 may be an emitter/detector configured to transmit and/orreceive IR, UV, or visible light signals, for example. In yet anotherembodiment, the transmit/receive element 1922 may be configured totransmit and/or receive both RF and light signals. It will beappreciated that the transmit/receive element 1922 may be configured totransmit and/or receive any combination of wireless signals.

Although the transmit/receive element 1922 is depicted in FIG. 19B as asingle element, the WTRU 1902 may include any number of transmit/receiveelements 1922. More specifically, the WTRU 1902 may employ MIMOtechnology. Thus, in one embodiment, the WTRU 1902 may include two ormore transmit/receive elements 1922 (e.g., multiple antennas) fortransmitting and receiving wireless signals over the air interface 1916.

The transceiver 1920 may be configured to modulate the signals that areto be transmitted by the transmit/receive element 1922 and to demodulatethe signals that are received by the transmit/receive element 1922. Asnoted above, the WTRU 1902 may have multi-mode capabilities. Thus, thetransceiver 1920 may include multiple transceivers for enabling the WTRU1902 to communicate via multiple RATs, such as NR and IEEE 802.11, forexample.

The processor 1918 of the WTRU 1902 may be coupled to, and may receiveuser input data from, the speaker/microphone 1924, the keypad 1926,and/or the display/touchpad 1928 (e.g., a liquid crystal display (LCD)display unit or organic light-emitting diode (OLED) display unit). Theprocessor 1918 may also output user data to the speaker/microphone 1924,the keypad 1926, and/or the display/touchpad 1928. In addition, theprocessor 1918 may access information from, and store data in, any typeof suitable memory, such as the non-removable memory 1930 and/or theremovable memory 1932. The non-removable memory 1930 may includerandom-access memory (RAM), read-only memory (ROM), a hard disk, or anyother type of memory storage device. The removable memory 1932 mayinclude a subscriber identity module (SIM) card, a memory stick, asecure digital (SD) memory card, and the like. In other embodiments, theprocessor 1918 may access information from, and store data in, memorythat is not physically located on the WTRU 1902, such as on a server ora home computer (not shown).

The processor 1918 may receive power from the power source 1934, and maybe configured to distribute and/or control the power to the othercomponents in the WTRU 1902. The power source 1934 may be any suitabledevice for powering the WTRU 1902. For example, the power source 1934may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd),nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion),etc.), solar cells, fuel cells, and the like.

The processor 1918 may also be coupled to the GPS chipset 1936, whichmay be configured to provide location information (e.g., longitude andlatitude) regarding the current location of the WTRU 1902. In additionto, or in lieu of, the information from the GPS chipset 1936, the WTRU102 may receive location information over the air interface 1916 from abase station (e.g., base stations 1914 a, 1914 b) and/or determine itslocation based on the timing of the signals being received from two ormore nearby base stations. It will be appreciated that the WTRU 1902 mayacquire location information by way of any suitablelocation-determination method while remaining consistent with anembodiment.

The processor 1918 may further be coupled to other peripherals 1938,which may include one or more software and/or hardware modules thatprovide additional features, functionality and/or wired or wirelessconnectivity. For example, the peripherals 1938 may include anaccelerometer, an e-compass, a satellite transceiver, a digital camera(for photographs and/or video), a universal serial bus (USB) port, avibration device, a television transceiver, a hands free headset, aBluetooth® module, a frequency modulated (FM) radio unit, a digitalmusic player, a media player, a video game player module, an Internetbrowser, a Virtual Reality and/or Augmented Reality (VR/AR) device, anactivity tracker, and the like. The peripherals 1938 may include one ormore sensors, the sensors may be one or more of a gyroscope, anaccelerometer, a hall effect sensor, a magnetometer, an orientationsensor, a proximity sensor, a temperature sensor, a time sensor; ageolocation sensor; an altimeter, a light sensor, a touch sensor, amagnetometer, a barometer, a gesture sensor, a biometric sensor, and/ora humidity sensor.

The WTRU 1902 may include a full duplex radio for which transmission andreception of some or all of the signals (e.g., associated withparticular subframes for both the UL (e.g., for transmission) anddownlink (e.g., for reception) may be concurrent and/or simultaneous.The full duplex radio may include an interference management unit toreduce and or substantially eliminate self-interference via eitherhardware (e.g., a choke) or signal processing via a processor (e.g., aseparate processor (not shown) or via processor 1918). In an embodiment,the WRTU 1902 may include a half-duplex radio for which transmission andreception of some or all of the signals (e.g., associated withparticular subframes for either the UL (e.g., for transmission) or thedownlink (e.g., for reception)).

As described above, systems and methods set forth herein provideembodiments that use gaze control to bring enhanced MR contentpertaining to faraway objects closer to the user. The solution uses avision guideline implemented as an MR object. The guideline containspoints that help the user to focus his gaze, placed at depths equivalentto the focally correct viewing distances supported by device hardware.The enhanced MR content follows the users gaze along the line, thusmoving the content closer or farther from the user. The location anddimensions of the line are determined by the system, based on HWrestrictions and existing real-life or MR objects in the user's view.

In accordance with an embodiment, a method includes forming athree-dimensional (3D) map of surroundings of a user of an augmentedreality (AR) head mounted display (HMD); determining a depth-wiselocation of a gaze point of a user based on eye gaze direction and eyevergence; determining a visual guidance line pathway in the 3D map;guiding an action of the user along the visual guidance line pathway atone or more identified focal points; and rendering a mixed reality (MR)object along the visual guidance line pathway at a locationcorresponding to a direction of the users gaze.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined by gaze tracking of the user.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined according to one or more hardwarerestrictions of the HMD.

In one or more embodiments, the one or more focal points along thevisual guidance line pathway are determined based on a number ofvisually correct focal planes displayable by the HMD.

In one or more embodiments, the one or more focal points along thevisual guidance line pathway are determined based on a determinedaccuracy of gaze tracking available on the HMD.

In one or more embodiments, the one or more focal points along thevisual guidance line pathway are determined based on a number ofvisually correct focal planes displayable by the HMD.

In one or more embodiments, the one or more focal points along thevisual guidance line pathway are determined based on a determinedaccuracy of gaze tracking available on the HMD.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined according to a distance of the MRobject.

In one embodiment, the one or more focal points along the visualguidance line pathway are determined by a movement of the user.

In one embodiment, the determining the visual guidance line pathway inthe 3D map is based on the depth-wise location of the gaze point of theuser and available space in the 3D map. In one embodiment, thedetermining the visual guidance line pathway in the 3D map includesforming the visual guidance line pathway to avoid one or more identifiedobjects in the 3D map.

In one embodiment, the determining the visual guidance line pathway inthe 3D map includes altering the visual guidance line pathway accordingto movements of the user, including one or more of a head tilt, a headpitch, a head yaw, and a gesture.

In one embodiment, the determining the visual guidance line pathway inthe 3D map includes altering the visual guidance line pathway accordingto one or more pivot points determined by a user gaze.

In one embodiment, the method also includes determining a number ofpoints along the visual guidance line pathway as a function of availablefocal planes in the 3D map.

Another embodiment is directed to a method including forming athree-dimensional (3D) map of surroundings of a user of an augmentedreality (AR) head mounted display (HMD); displaying a mixed reality (MR)object in the 3D map including a visual cue that content enhancement isavailable to the user for the object; activating the content enhancementaccording to user input to the HMD with respect to the visual cue;displaying a visual guidance line pathway in the 3D map; guiding anaction of the user along the visual guidance line pathway at one or moreidentified focal points; and rendering the MR object along the visualguidance line pathway at a location corresponding to a direction of theuser's gaze.

In one or more embodiments, the activating the content enhancementaccording to user input to the HMD includes user input of one or more ofa gaze, a head gesture, or a gaze dwelling on the content enhancement.

In one or more embodiments, the displaying the visual guidance linepathway in the 3D map includes displaying the one or more identifiedfocal points as a plurality of focal plane indicators at a plurality ofdepths within the 3D map. In one or more embodiments, the rendering theMR object along the visual guidance line pathway at the locationcorresponding to the direction of the users gaze includes moving theenhancement object along the plurality of focal plane indicators at theplurality of depths to enlarge the MR object.

In one or more embodiments, the guiding the action of the user along thevisual guidance line pathway at the one or more identified focal pointsincludes providing the visual cue, wherein the visual cue includes anext suggested action for the user.

In one or more embodiments, the displaying the visual guidance linepathway in the 3D map includes determining a depth-wise location of agaze point of a user based on eye gaze direction and eye vergence.

Another embodiment is directed to a system including a processor and anon-transitory computer-readable storage medium storing instructionsoperative, when executed on the processor, to perform functionsincluding forming a three-dimensional (3D) map of surroundings of a userof an augmented reality (AR) head mounted display (HMD); determining adepth-wise location of a gaze point of a user based on eye gazedirection and eye vergence; determining a visual guidance line pathwayin the 3D map; guiding an action of the user along the visual guidanceline pathway at one or more identified focal points; and rendering amixed reality (MR) object along the visual guidance line pathway at alocation corresponding to a direction of the user's gaze.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined by gaze trackingof a user.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined according to oneor more hardware restrictions of the HMD.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined according to adistance of the MR object.

In one or more embodiments of the system, the one or more focal pointsalong the visual guidance line pathway are determined by a movement ofthe user.

In one or more embodiments of the system, the determining the visualguidance line pathway in the 3D map is based on the depth-wise locationof the gaze point of the user and available space in the 3D map.

In one or more embodiments of the system, the determining the visualguidance line pathway in the 3D map includes forming the visual guidanceline pathway to avoid one or more identified objects in the 3D map. Inone or more embodiments of the system, the determining the visualguidance line pathway in the 3D map includes altering the visualguidance line pathway according to movements of the user, including oneor more of a head tilt, a head pitch, a head yaw, and a gesture.

In one or more embodiments of the system, the determining the visualguidance line pathway in the 3D map includes altering the visualguidance line pathway according to pivot points determined by user gaze.

Another embodiment of the system is directed to the non-transitorycomputer-readable storage medium storing instructions operative, whenexecuted on the processor, to perform additional functions includingdetermining a number of points along the visual guidance line pathway asa function of available focal planes in the 3D map.

Another embodiment is directed to a system including a processor and anon-transitory computer-readable storage medium storing instructionsoperative, when executed on the processor, to perform functionsincluding forming a three-dimensional (3D) map of surroundings of a userof an augmented reality (AR) head mounted display (HMD); displaying amixed reality (MR) object in the 3D map including a visual cue thatcontent enhancement is available to the user for the object; activatingthe content enhancement according to user input to the HMD with respectto the visual cue; displaying a visual guidance line pathway in the 3Dmap; guiding an action of the user along the visual guidance linepathway at one or more identified focal points; and rendering the MRobject along the visual guidance line pathway at a locationcorresponding to a direction of the user's gaze.

In one or more embodiments of the system, the activating the contentenhancement according to user input to the HMD includes user input ofone or more of a gaze, a head gesture, or a gaze dwelling on the contentenhancement.

In one or more embodiments of the system, the displaying the visualguidance line pathway in the 3D map includes displaying the one or moreidentified focal points as a plurality of focal plane indicators at aplurality of depths within the 3D map.

In one or more embodiments of the system, the rendering the MR objectalong the visual guidance line pathway at the location corresponding tothe direction of the user's gaze includes moving the enhancement objectalong the plurality of focal plane indicators at the plurality of depthsto enlarge the MR object.

In one or more embodiments of the system, the guiding the action of theuser along the visual guidance line pathway at the one or moreidentified focal points includes providing the visual cue, wherein thevisual cue includes a next suggested action for the user.

In one or more embodiments of the system, the displaying the visualguidance line pathway in the 3D map includes determining a depth-wiselocation of a gaze point of a user based on eye gaze direction and eyevergence.

Another embodiment is directed to a method for rendering a visualguidance pathway including forming a three-dimensional (3D) map ofsurroundings of a user of an augmented reality (AR) head mounted display(HMD); determining a depth-wise location of a gaze point of a user basedon eye gaze direction and eye vergence; determining a visual guidanceline pathway in the 3D map; and rendering one or more mixed reality (MR)objects along the visual guidance line pathway at locationscorresponding to a direction of the user's gaze, while avoiding one ormore preexisting objects in the 3D map of the surroundings.

In one or more embodiments of the method, the visual guidance linepathway is placed in determined available space within the 3D map of thesurroundings.

In one or more embodiments of the method, the one or more preexistingobjects include one or more real-world objects and existing MR objects.

What is claimed is:
 1. A method comprising: forming a three-dimensional(3D) map of surroundings of a user of an augmented reality (AR) headmounted display (HMD); selecting a mixed reality (MR) object forplacement or repositioning within the 3D map; displaying a visualguidance line pathway in the 3D map; detecting a gaze of the user at oneor more positions along the visual guidance line pathway; and renderingthe MR object at rendering positions along the visual guidance linepathway corresponding to the one or more detected positions of the gazeof the user.
 2. The method of claim 1, further comprising: determiningthe visual guidance line pathway in the 3D map; and determining the gazeof the user at the one or more positions along the visual guidance linepathway, wherein determining the gaze of the user at the one or morepositions along the visual guidance line pathway comprises any one of:determining the one or more positions by gaze tracking of the user,determining the one or more positions according to one or more hardwarerestrictions of the AR HMD, determining the one or more positions basedon a number of visually correct focal planes capable of being displayedby the AR HMD, determining the one or more positions based on adetermined accuracy of gaze tracking available on the AR HMD,determining the one or more positions according to a distance of the MRobject, or determining the one or more positions by a movement of theuser.
 3. The method of claim 2, wherein determining the visual guidanceline pathway in the 3D map comprises any one of: determining the visualguidance line pathway based on a depth-wise location of a gaze point ofthe user and available space in the 3D map, determining the visualguidance line pathway by forming the visual guidance line pathway toavoid one or more identified objects in the 3D map, determining thevisual guidance line pathway by altering the visual guidance linepathway according to pivot points determined by a user gaze, ordetermining the visual guidance line pathway by altering the visualguidance line pathway according to movements of the user, the movementsof the user comprising one or more of a head tilt, a head pitch, a headyaw, and a gesture.
 4. The method of claim 1, wherein the visualguidance line pathway is a straight line occupying multiple depthsrelative to the user.
 5. The method of claim 1, wherein the visualguidance line pathway is a curved line displayed in 3D space.
 6. Themethod of claim 1, wherein the visual guidance line pathway is displayedwith multiple point markers, the multiple point markers identifying therendering positions at which the MR object is available to be rendered.7. The method of claim 1, wherein the multiple point markers correspondto discrete focal planes supported by an optical system of the HMD. 8.The method of claim 1, further comprising determining the visualguidance line pathway to avoid objects in the 3D map.
 9. The method ofclaim 1, further comprising determining the visual guidance line pathwayto avoid additional MR objects displayed for the user by the HMD. 10.The method of claim 1, wherein selecting the MR object comprises:identifying an object in a view of the user for which contentenhancement is available; and selecting an available content enhancementobject for the identified object to be the selected MR object.
 11. Themethod of claim 1, further comprising: detecting a gaze point of theuser, wherein selecting the MR object is based on a detected gaze pointof the user.
 12. The method of claim 1, wherein rendering the MR objectat the rendering positions along the visual guidance line pathwaychanges a depth at which the MR object is rendered relative to the user.13. The method of claim 1, further comprising: displaying a visual cuethat a content enhancement is available to the user for an objectcorresponding to the MR object, wherein selecting the MR objectcomprises activating content enhancement for the object.
 14. The methodof claim 1, wherein rendering the MR object further comprises: renderingone or more MR objects along the visual guidance line pathway atlocations corresponding to a direction of the gaze of the user, whileavoiding one or more preexisting objects in the 3D map of thesurroundings, wherein the visual guidance line pathway is placed indetermined available space within the 3D map of the surroundings, andwherein the one or more preexisting objects comprise at least one of areal-world object or an existing MR object.
 15. A system comprising: aprocessor; and a non-transitory computer-readable storage medium storinginstructions operative, when executed by the processor, to cause thesystem to: form a three-dimensional (3D) map of surroundings of a userof an augmented reality (AR) head mounted display (HMD); select a mixedreality (MR) object for placement or repositioning within the 3D map;display a visual guidance line pathway in the 3D map; detect a gaze ofthe user at one or more positions along the visual guidance linepathway; and render the MR object at rendering positions along thevisual guidance line pathway corresponding to the one or more detectedpositions of the gaze of the user.
 16. The system of claim 15, whereindisplaying the visual guidance line pathway in the 3D map comprises:displaying the one or more identified focal points as a plurality offocal plane indicators at a plurality of depths within the 3D map. 17.The system of claim 15, wherein the instructions are further operative,when executed by the processor, to cause the system to: display a visualcue that a content enhancement is available to the user for an objectcorresponding to the MR object, wherein activating the MR objectcomprises activating the content enhancement for the object.
 18. Thesystem of claim 15, wherein the instructions are further operative, whenexecuted by the processor, to cause the system to: prior to activating,display the object corresponding to the MR object in the 3D map alongwith the visual cue.
 19. The system of claim 15, wherein the objectcorresponding to the MR object comprises one of a virtual object, apre-existing real world object, or an MR object.
 20. The system of claim17, wherein activating the MR object further comprises activating thecontent enhancement for the object corresponding to the MR objectaccording to the user input to the AR HMD with respect to the visualcue.